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INVARIANT TRILINEAR FORMS FOR SPHERICAL 
DEGENERATE PRINCIPAL SERIES OF COMPLEX 
SYMPLECTIC GROUPS 

PIERRE CLARE 


Abstract. We compute generalized Bernstein-Reznikov integrals as¬ 
sociated with standard complex symplectic forms by studying Knapp- 
Stein intertwining operators between spherical degenerate principal 
series of Sp(n, C). 


Introduction 

Let (7ri,Ui), (7r2, V2) and (^3,1/3) be continous complex representations 
of a topological group G. An invariant trilinear form is a trilinear map 
d' : Ui X V2 X U3 — > C such that 

d' (vri(5f) • Vi,'K 2 {g) ■ V2,'Kz{g) ■ V 3 ) = ^{Vi,V 2 ,V 3 ) 

for all g E G and Vi G U- 

Interest for invariant trilinear forms attached to representations of Lie 
groups was sparked by the study of fusion rules, that is, the spectral de¬ 
composition of tensor product of irreducible representations. See [Oks73, 
Rep78, M0I79, LokOl] for results in the case of SL(2,M). More gener¬ 
ally, the understanding of these forms, in particular of their linear depen¬ 
dence relations, is involved in the study of multiplicities in branching laws 
[K013]. They are also known to play an important role in the study of 
composition formulas in quantization theory. See for instance [Unt03] and 
[K0PUO9]. 

Invariant trilinear forms appeared in connection with automorphic rep¬ 
resentation theory in the work of J. Bernstein and A. Reznikov, who used 
them in [BR04] to obtain estimates of automorphic forms. Their result 
relies on the asymptotic study of certain integral on S^ x S^ x S^ for which 
a closed formula is given in terms of the Euler’s function T. It corresponds 
to the evaluation of the essentially unique invariant trilinear form associ¬ 
ated with spherical principal series of PGL(2, M) on triples of PO(2)-hxed 
vectors. 
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This setting was generalized to spherical principal series representations 
of more general families of Lie groups: invariant trilinear forms associated 
with these representations give rise to integrals on triple flag varieties 
when evaluated on iL-fixed vectors. Such integrals are called generalized 
Bernstein-Reznikov integrals. 

The case of Lorentz groups SOo(d, 1) was studied separately in [Dei06] 
and [CK0P11]. Closed formulas for other generalized Bernstein-Reznikov 
integrals were obtained in [CK0P11] in relation with the representation 
theory of different families of groups, namely Sp(n, M) and GL(n, M). The 
method of [CK0P11] also allowed to recover the case of Lorentz groups 
from another point of view. A complementary study of these conformally 
invariant trilinear forms was done in [BC12] where residues at singular 
parameters were computed. 

Finally, a systematic study of invariant trilinear forms constructed from 
Knapp-Stein intertwiners attached to spherical principal series represen¬ 
tations of semisimple Lie groups of real-rank 1 groups was achieved in 
[BKZ14], where a description of the residues at some poles in terms of 
bidifferential operators is obtained. Note also that generalizations to sym¬ 
metric pairs are considered in [KS14] and [KS15], with different goals. 

The general framework in which these integrals are related to Represen¬ 
tation Theory is described in Section 5 of [CK0P11], where the important 
role played by Knapp-Stein intertwiners and their normalisations is high¬ 
lighted: generalized Bernstein-Reznikov integrals are interpreted as traces, 
which can in turn be evaluated when these operators are understood in 
sufficient detail. More precisely, the heart of the computation of these 
traces is the explicit determination of the iL-spectrum of the intertwiners. 

The work presented here follows this method. It relies on results about 
degenerate principal series of Sp(n, C) obtained in [Clal2], where normal¬ 
ized Knapp-Stein intertwiners were described in terms of complex sym- 
plectic Fourier transformations, leading to the iF-type analysis needed to 
evaluate the corresponding generalized Bernstein-Reznikov integrals. 

Main result. The purpose of this note is to study, for n > 1 integer, the 
following integral: 

X„ = J \Reoj{y, z)\'^~'^ \Reoj{z,x)\^~^ \Reu;{x,y)\'^~'^ da{x,y, z) 

g4n—1 ^g47i —1 ^g4n —1 

where u denotes the standard complex sympletic form on C^"' and a is the 
cube of the euclidean measure crs4n-i on the (4n — l)-dimensional sphere: 

(T ^ (Tg4n —1 (Tg4n —1 (Tg4n —1. 

More precisely, we establish the following result. 
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Theorem 1 . As a meromorphic function of the parameters a, f 5 and 7, 
the integral is equal to 


with 


2 n — 1, 2n — 1, n + ^ , 

1 , n-I, 


2n—a 
4 ’ 

6n+a 
4 ’ 


2n—/3 2n—7 

4 ’ 4 ’ 

6n+/3 6n+7 

4 ’ 4 ’ 


A{a,P,-f) 


■p /^2—27^^-Q^ p /^2—2nH-/3'\ p /^2—2n+7^ 
o 6n-f M 4 JM 4 JM 4 J 
p ^ 6nH-Q ^ p ^ 6nH-/3 ^ p ^ 6nH-7 ^ 




The definition of the hypergeonietric function pFq is recalled p.ll and 
the precise domain of convergence for X„ is given by Proposition 4.1. 


Outline. Notations and definitions are fixed in Section 1, where we also 
recall useful facts about real, complex and quaternionic spherical harmon¬ 
ics. Sections 2 and 3 are devoted to the discussion of spherical degenerate 
principal series of Sp(n, C) and the corresponding Knapp-Stein intertwin¬ 
ers. In particular, the iP-spectrum of these operators is given in Proposi¬ 
tion 3.1. Theorem 1 is proved in Section 4 and the domain of convergence 
is determined in Proposition 4.1. Finally, we use classical and more recent 
reduction results on well-poised hypergeometric functions to derive closed 
formulas when n = 1 and n = 2 in Section 5. 


1. Preliminaries 

The representation theory of complex symplectic groups is naturally 
formulated in terms of quaternionic and complex vector spaces. However, 
it will often be convenient to deal with functions of real variables and their 
integral transforms. The next paragraphs are devoted to fixing identifica¬ 
tions and introducing the relevant functional spaces and transformations 
to be used in the proof of the main result. 

1.1. Identifications and bilinear forms. Let M, C and El be the fields 
of real, complex and quaternionic numbers respectively, i and j the usual 
anticommuting complex and quaternionic units. For any integer n > 1, 
we fix the following identifications: 

(1) H” ~ (C’* + jC^) ~ ((R^ + + j (R^ + *^)). 

The identity matrix on KP, with K any of the above fields, is denoted 
by Ip. The complex conjugation on = R^" -|- iR^"' ~ R^" x R^" is 
implemented by the matrix; 


hn 

0 

0 

1 

7" 

3 
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and if / is a function on we denote by the function f{s-). 

For X = {xi,, X 2 n) and F = (i/i,..., i/ 2 n) in denote 

2n 

{X,Y) = ^XkVk- 

k=l 

The herniitian inner product on is thus dehned by 
(2) (X,F) = (X,F) = (Xk,£Fm) 

where Xr and Fr are the images of X and F under the identihcation 

£^2n ^ ]^4n gjygj^ Py 

The symplectic structure on is given by the matrix 


0 

1- 

1 

[ In 

o 

1_ 


corresponding to the element j G H. 

The symplectic form u is dehned by a;(X, F) = (X, JY), that is, 

n;(X,F) = (X2,Fi)-(Xi,F2). 

The complex symplectic group is the group of complex invertible matrices 
of size 2n preserving ca: 

Sp(n,C) = {(/e GL(2n,C) |VX,FeC2", u{gX,gY) = u{X,Y)] . 

Equivalently, the elements g of GL(2n, C) belonging to Sp(n, C) are ex¬ 
actly those that satisfy the relation ^gJg = J. 

1.2. Integral transforms. Most of the computations in this paper rely 
on the use of integral transformations, similar to the classical Fourier 
transform on euclidean spaces. Here, we collect the various operators 
that will be used and the relations between them. The formulas initially 
make sense for Schwartz functions and extend to operators on the space 
of square-integrable functions. 

Consider the complex Fourier transform on 

-^c-/(0= [ 

It allows to dehne the complex symplectic Fourier transform by 

(3) -^symp/(0=-^C-/(^0, 

that is, 

^symp/(0 = f dX. 

The real conjugate Fourier transform on is dehned by 

(4) 
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SO that 

In other terms, 

/(O = f d(Xi, X2). 

As explained above, nnder the identihcation Rf” X R 2 "' ~ given by 
(Xi, X 2 ) I —> Xi + iX 2 i the matrix £ induces the complex conjugation. By 
(2), the complex Fourier transform can be seen as the transform Xs f = 
(XR4n jy. Indeed, if ^ = (^ 1 ,^ 2 ) and C = + i^ 2 , h is clear that 


-^c-/(C) = ^./(0- 

Therefore, the complex symplectic Fourier transform (3) is related to 
the real conjugate Fourier transform (4) by the following formula: 

(5) J^sympf = 

According to the above dehnitions, the left-hand side of this identity is a 
function of a complex variable, whereas the one on the right is a function of 
a real variable. This ambiguity is resolved by the identihcations (1), which 
also allow the multiplication by the quaternionic unit j. The fact that this 
multiplication occurs on the right although the dehnition of Xsymp involves 
the matrix J acting from the left is due to the complex conjugation and 
the relation jz = zj, true for any complex number 


1.3. Spherical harmonics. The iF-type analysis of the representations 
involved in our computations mainly relies on the decomposition of square- 
integrable functions on the sphere Under the identihcations of (1), 

can be seen as the unit sphere Si in H”, or Moreover, these 
identihcations are compatible with the standard actions of the orthogonal, 
unitary and symplectic groups, as well as that of scalars of modulus one, 
namely {±1} C R, U(l) C C and Sp(l) C H. Note that we let scalars act 
on the right, in order to avoid confusion in the non-commutative quater¬ 
nionic case. 

The situation is summarized in the following diagram and the relevant 
quotients shall be described in Section 2. 


Sp(n) 

rA SyiT) 

Sp(l) 

n 

l\ 

U 

U(2n) 


^ U(l) 

n 

l\ 

u 

0(4n) 

rx 5i(R^’^) 
d 

g4n —1 

{if} 


~ SU(2) 
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The compatibility of the identihcations allows us to consider functions 
on the sphere as depending on real, complex or quaternionic variables as 
needed. With this general picture in mind, let us recall some facts that 
will be of use further on. Proofs and details can be found in [CK0P11] and 
[Clal2]. Following classical notations, we denote by the space of 

harmonic homogeneous polynomials of degree k on which is naturally 
a representation of 0(4n). Analogously, for a and (3 integers, 
will denote the space of polynomials P{Z, Z) on that are homogeneous 
of degree a in Z = (zi,..., Z 2 n), of degree /3 in Z = (zi,..., Z 2 n) and 
subject to the condition 

2 ? 7 . r\0 

y =0. 

dzfdzf 

By dehnition, there is a natural isomorphism of vector spaces that com¬ 
mutes to the U(2?7,)-actions: 

(6) |u(2n) ^ 0 

a+j3=k 

In the quaternionic case, we denote by V^' the unique unitary ir¬ 
reducible representation of Sp(n) corresponding to the highest weight 
(/, 0,..., 0) where I and I' are integers satisfying I > I' > 0. Similarly, 0/ 

denotes the irreducible j-I-1-dimensional representation of Sp(l) ~ SU(2). 
Analogously to (6), one has the following compatibility relation between 
representations of Sp(n) and U(2n): 

(7) |sp(„) ^ 0 q-''. 


Since U(l) naturally embeds into SU(2) via 


e 


id 




e*® 0 

0 e-*® ’ 


Vl decomposes according to the characters of U(l). More precisely, if C 5 
denotes the space of the character z ^ z^ for 5 G Z, then 

(8) n ~ 0 c,. 

I'5|<i 
<5=1 [2] 

The isotypic decomposition of with respect to the action of the 

direct product Sp(n) x Sp(l) is given in [HT93]; 

L2 (S477-1) ^ yhV ^ Y^-i\ 

l>l'>0 
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Together with ( 8 ), this result leads to the following decomposition under 
the action of Sp(n) x U(l): 


(9) ~ ® Q. 


2. Spherical degenerate principal series and their K-types 


The complex symplectic group G = Sp(n, C) acts on the projective 
space The stabiliser of a point under this action is a maximal 

parabolic subgroup 

P = LiV ~ (C^.Sp(m,C)) X 

where m = n — 1. For any complex number A, the character a i—)■ |a|^ of 
can be extended to a character of -P, then induced to G: 

tta = IndpXA- 


The family {TAlAgjR is called the spherical degenerate principal series of 
Sp(u,C), and was investigated in [Gro71] and [Clal2]. In particular, tta 
is irreducible for A 7 ^ 0, representations with opposite parameters tta and 
7 r_A are equivalent and tq splits into the direct sum of two irreducible 
components. We recall here more precise facts about these representations 
and the intertwining relations between them that will be of later use. 

The representation tta can be realised on a space that admits 

= [f e G^{C^^ \ {0}) 1 Va e C^ /(a •) = } 

as a dense subspace. By homogeneity, functions in are determined by 
their restriction to the euclidean sphere The compact picture of tta 

is obtained by restriction to 8 ^”“^ according to the identideations 

K/{L n iF) ~ 8 p(n)/U(l). 8 p(m) ~ 8 ^”-VU(l) 


and tta is realised on ( 8 '^"'“^). Using quaternionic spherical harmonics 
as described in Preliminaries 1.3, more precisely hxing 5 = 0 in (9), this 
leads to the iP-type decomposition 


( 10 ) 


^^lsp(n) 



l>l'>0 
1 = 1 '[ 2 ] 


Remark 2.1. This formula is a special case of the iP-type decomposition 
of non-spherical degenerate principal series of 8 p(n, C) given in [Clal 2 , 
Proposition 2]. 
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3. Intertwining operators 

Intertwining operators among the degenerate principal series of Sp(n, C) 
were studied in [Clal2] , where normalization was obtained by means of the 
complex symplectic Fourier transform and the iF-spectrum was explicitely 
computed. Let us recall here the facts that will enter the determination 
of invariant trilinear forms. 

3.1. Normalization of the Knapp-Stein intertwiners. If / is a square- 
integrable function / on and Re(A) < —2n + 1, the integral 

Txf{y) = [ f{x) |Re(n;(a;,i/))r^"^'' da{x) 

ys4^-i 

is absolutely convergent, thus yielding a meromorphic family {T^} of oper¬ 
ators that are easily seen to intertwine the actions of Sp(n, C) on 
seen as the compact picture for 7i_x and tix respectively. Therefore, Tx is 
called the non-normalized Knapp-Stein intertwiner between 'k_x and ttx- 
The corresponding normalized intertwiner is given by the symplectic 
Fourier transform of Preliminaries 1.2. More precisely, it is proved in 
[Clal2, Proposition 1] that: 


( 11 ) 


Ta = 271 


A+2n— 


.r 


¥-») 


r(l + ") 


T, 


symp 




3.2. iP-spectrum of Tx- Each summand occurs with multiplicity 1 in the 
Lf-type formula (10) so the intertwiner Tx acts on each by a scalar. 
The following result specifies the eigenvalues in this ‘diagonalisation’. 


Proposition 3.1. The Knapp-Stein intertwiner Tx acts on by 


2^1+1'W 


271^” 2 


-T(¥-")r( 


r(l + ") r( 


;+;'+A 

2 

l+l'-X 

2 


+ n) 
+ n) 


Proof. In view of the compatibility between the isotypic decompositions 
of Preliminaries 1.3, an element of can be seen as the restriction to 
the unit sphere of a polynomial in 

Consider p in Its restriction p|g 4 n-i can be extended to a 

homogeneous function p^ on by 


Pairu) = r°‘p{u). 

The image of Pa under the Fourier transform is given by the following 
identity between distributions on R^" depending meromorphically on a, 
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proved in [CK0P11, Lemma 2.3]: 

J ~ ]]j4n Pq, TT Z ^ ^ k — a ^ P — a — 4 n - 

Using the expression of Tx in terms of the complex symplectic Fourier 
transform (11) and the relation between the latter and the real Fourier 
transform (5) on letting a = X — 2n and A: = / + /' in (12) leads to; 


TxPx-2n = 271 " H 


r(iii-«)r(!±|±i + n) 


P—(X—2n)i'j ) 


F(| + n) F(^ + n)- 
The action of the right multiplication by j in the last formula can be 


determined by seeing V^' as a subspace of "H ^ ^ (C^”), according to 

Preliminaries 1.3. More precisely, the discussion of Section 6.3 of [HT93] 


l-V _ _ 

implies that the polynomial {ziWi) ^ [z 2 W\ — Z 1 W 2 ) is a highest weight 

vector in . It implies that Px- 2 n{-j) = (“1)^ P\- 2 n for any p in 
and the result follows. □ 


4. Triple integrals 

Let us now turn to the proof of Theorem 1. Following [CK0P11], our 
strategy will be to interpret X„ as the trace of an operator build from the 
intertwiners discussed above. For j G {1,2,3}, we let 

+ A 2 + A 3 — 2n 
Pj ~ 2 

and denote by p the triple p 2 , Ps)- 

Each is a Hilbert-Schmidt operator on ( 8 ^*^“^) for Re/ij << 0, 
so the composition is of trace class and its trace is given by 

Tn = Trace (T^, ) 

Reuj{y, z)\~^^~‘^"' |Rea;( 2 :, |Rea;(x, da^{x, 

(S4"-l)3 

where da^{x, y, z) denotes da(x) da(y) da(z). 

The trace can be computed on the convergence domain, that will be 
made precise in Paragraph 4.2, by means of the ‘diagonalisation’ of the 
previous section. 

4.1. Proof of the main result. According to the iP-type formula (10) 
and Proposition 3.1, one has 
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and reorganising the sum with respect to the integer m = ^ gives 


(13) 


Observe that 


m>0 Vj=l l+l'=2m 


^2»n(/l') — 


r (n + m + ^) 


(n + m- f) 

with Kn,fj. independent of m, so that 

^2m+2(/i) _r(n + m + l + |)r(n + m-|) _ n + m + 


^2m(/l) r^n + lTl+l — + 77. + 771 

Using the Pochhammer symbol 


, , r(a + r) 

= —7T73— = + 1)... (a + r — 1), 


it follows that 
(14) 
with 
( 16 ) 


r(a) 


^2m 


n 


+ l) 

2 / 7 


77 


2^7 


■^o(/i) 


Alo(p) = 




r(«-f) ■ 

The isomorphism (7) implies that 

dim Vy = dim 77™’™ . 

The right-hand side is given by Formula (3.3) in [CK0P11], and it follows 
that 

(2/77 + 2n — 1) ((m + l) 2 „- 2 )^ 




277-1 

The duplication formula for P implies 


r(2n)r(2r7 - 1) 

2777 277 - 1 f {m+ l)2n-2 

r(2r7 - 1) 


(a)orr,. = 2^™ 


a 

2/7n 


C7 -|- 1 


2m + 2n — 1 
277 -1 


(27i: 


2m 


(277 - l)2m’ 


SO observing that 


tol'trltol'?: 
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we get 

(17) 


2m + 2n — 1 


2 n - 1 - 5 ) ■ 

To deal with the second factor in (16), it is enough to observe that 
(m + l) 2 n -2 _ r(m + 2n - 1) _ {2n - 1)^ 

r(2n — 1) r(m + l)r(2n — 1) r(m + 1) 

The definitions imply that r(m + 1) = m! = (l)m, so Formulas (17) and 

(18) lead to 

(19) (2"-l)».(2"-l)».h + 5), 


dim IT 


l+r=2m 

The discussion above allows to write as the sum of a hypergeometric 


series. 


Definition 4.1. If p and q are positive integers, the hypergeometric func¬ 
tion with parameters ai,... ,ap and hi,... ,hg is defined by 

\ (fli)fc(fl2)fc • • • (op)fc 2 :^ 

j ^ {bi)k{b2)k ■ ■ ■ {bq)k k\ 

Then, (13), (14), (15) and (19) imply that 


(20) 

( 2n — 1 , 2n — 1 , n-fl. 

n+f , 

n + ^ , 

n - 1 - ^ 

\ 

with 


n-f , 

n - ^ 
n 2 5 

n-f 


~ ^o(hi)24o(h2)^o(h3) 

= r - ») r - ») r - ») 

r(n-f) r(n-f) r(n-f) ’ 

and Theorem 1 follows from letting a = Ai — A2 — A3 = — 2 (n + pi), 
f3 = —Ai + A 2 — A 3 = — 2{n + P 2 ) and y = —Ai — A 2 + A 3 = — 2{n + t^f)- 

4.2. Domain of convergence. The integrals discussed above are origi¬ 
nally defined for parameters (pi, /i 2 , Ps) in regions that guarantee absolute 
convergence, then extended meromorphically with respect to these param¬ 
eters. In order to give an exact description of the convergence region, the 
method used in [CK0P11] can be directly applied. In order to use real 
coordinates, we identify and via 

{Xl + tXl xl + tXl) ^ (Xj, xl xl) 
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and let 


■ J 

0 

0 

-J 


One then gets, for X,Y G 

Rea;(X,F) = {X, $y), 

and, considering 

h^{X,Y,Z) = |(F,d>Z)|"^ |(Z,$X)|"^ |(X,d>F)|"^ 

with A = (Ai,A 2 ,A 3 ) and replacing J by $ everywhere in the proof of 
Proposition 6.2 of [CK0P11] directly leads to the following; 

Proposition 4.1. The integral 

/ia(X, F, Z) da{X) da{Y) da{Z) 

g4n — 1 g4n — 1 ^ S^n — 1 

converges absolutely if and only i/Re(Aj) > —1 for j G {1,2,3}. 

5. Closed formulas 

The equality (20) allows to express the generalized Bernstein-Reznikov 
integral in terms of a hypergeometric function, namely of the form 
0 X 5 . Under certain conditions on the parameters, such functions are said 
well-poised and special values can sometimes be evaluated as quotients of 
expressions involving Euler’s function P. 

Such formulas have been obtained in the conformal case in [Dei06], 
[C011], [CK0P11] and [BKZ14] and further results were recently estab¬ 
lished by J.-L. Clerc in [Clel4, ClelS]. Similarly, closed formulas were 
found in the case of real symplectic groups in [CK0P11], where the meth¬ 
ods used here find their origin. See also the composition formulas obtained 
in pseudo-differential calculus related to the quantization of certain ho¬ 
mogeneous spaces [PU07, Unt03]. 

On the contrary, the Euclidean case, corresponding to the real gen¬ 
eral linear group studied in [CK0P11] and the complex, quaternionic and 
octonionic real-rank 1 cases discussed in [BKZ14] give rise to sums of gen¬ 
eralized hyper geometric series for which no closed formulas are currently 
available. 

Beyond the prospect of discovering new reduction formulas for hyper¬ 
geometric functions, the interpretation of the existence of closed formulas 
for generalized Bernstein-Reznikov integrals (or lack thereof) in geomet¬ 
ric terms is an interesting problem. Aspects of this problem, such as the 
generic uniqueness of invariant trilinear forms are discussed in Section 5.1 
of [CK0P11]. It is related to the study of orbits of the diagonal action 
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of the group G on certain triple flag varieties G/P x G/P x G/P which 
generalize the product of circles that appears in the original work of Bern¬ 
stein and Reznikov. See the introduction of [CN06] and [Clel4, Clel5] for 
a review of known results and extensions of these ideas to a broader con¬ 
text, such as that of the Shilov boundary of certain symmetric domains 
[Cle07]. 

In the remainder of this paragraph, we derive and discuss closed formu¬ 
las in the complex symplectic case for n = 1 and n = 2. 


5.1. Case n = 1. Let us go back to the general formula (13) giving as 

ies m = I and 

dim Vy = dim = 2m + 1, 


a trace. Fixing I' = 0 implies m = ^ and 


so (13) becomes 


/I - (1 h- ^ ^ ^ 

m>0 V 2 Jm\ 2Jm\ 2)m 


with 


— ^o(/^l)^o(/^2)^o(/^3) — 8^^ 


The final form of the result is 


r (-ijyi) r (-ap) r (- 1 ^) 
r (1 - f) r (1 - f) r (1 - f) 


1 3 1 Ml 1 _l_ M2 1 M3 

-*-) 2’ 2’ 2’ 2 
Ti = 5 F 4 I ; 1 


1 1 _ Ml 1 _ M2. 1 _ M3 

2’ 2 ’ 2 ’ 2 


A closed formula can be obtained for ti by means of the Dougall- 
Ramanujan formula given in [CK0P11, p.416] (see also [Bai35, pp.25-26]), 
namely 


m — 1, 


( 21 ) 


m+l 
2 ’ 

m—1 


—X 


-y , 


5i"4| ;1 

2 ’, X + m , y + m, z + m 

r(a; -|- m)V{y + m)T{z + m)T{x + y + z + m) 
r(m)r(a; + y + rn)T{y + z + m)T{x + z + m) 


Letting m = 2, x = —1 — y = —1 — ^ and z = —1 — ^ in (21) 
yields: 


(t) n = ( 2712 ) r 


Pi — ^2 — ys — ‘2 


r (-^ - f) 

LI Y ^ Mj-Ml-M2-M3 ^ 


n 
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5.2. Case n = 2. In recent work, K. Fatawat and V. Yashoverdhan 
[VF14] have extended Bailey’s classical method to study generalized hy¬ 
pergeometric functions under certain compatibility conditions on the pa¬ 
rameters. Of particular relevance here is the following generalization of 
Dougall’s theorem (see Formula (4.12) in [VF14] for the proof and [Bai35] 
or [Sla66] for Dougall’s original result); 


/-I, i + 1, ^ 


6-r’5 


( 22 ) 


2 ' 2 ’ 
/_1 


U 


V 


w 




2 

F(/ - u)T{f - v)T{f - n;)F(/ - ^- ^;-^;-l) 
F(/)F(/ - n - w)T{f -u-v- 1)F(/ - u - w - 1) 


K 


where 


K = 


vw — h{l + U + V + W — f) 


and h = 


/(/- 2 ) 


h{l + u + V — f){l + u + w — f) 4m 

Remark 5.1. Note that our k is the inverse of the quantity k in [VF14]. 
Setting the parameters / = 2n = 4 and 


2n — a 


u = 


V = 


2n- 13 


w = 


2 n — 7 


in (22), so that h = 


4 — a 


and 


_ (4 - a)(4 - /9)(4 - 7 ) 32(a + /? + 7 ) 

8(4-f a + /3)(4-f a + 7 ) 

we obtain 

47riO0F F 

^2 - ^ F (2 + ^) F (1 + ^) F (1 + ^) ' 

that is. 


((4 - a)(4 - /3)(4 - 7 ) + 32(a + ;3 + 7 )) T {^) 

6 (4 + a + ,3)(4 + a + ^)r(2 + to)r(l + 74)r(l + 72 )’ 

which reduces to 

, , ^ ((4 - a)(4 - /3)(4 - 7 ) + 32(a + fi + j))T (g±f±^) 

* 96 r (2 + to) r (2 + to) r (2 + to) 
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5.3. Concluding remarks. The construction of an invariant trilinear 
form obtained above by taking the trace of compositions of Knapp-Stein 
intertwiners follows the method used in the work presented in the intro¬ 
duction, in particular [CK0P11]. As in the situations that were previously 
studied, the Bernstein-Reznikov integrals associated with spherical degen¬ 
erate principal series of the complex symplectic group Sp(n, C) is can be 
expressed in terms of hypergeometric functions. 

The existence of the closed formula (f) in the case n = 1 is consistent 
with the results obtained in the conformal case studied in [CK0P11], 
[C011] and [Dei06], in view of the isogeny between Sp(l,C) = SL(2,C) 
and SOo(3, 1). 

The proof of [Dei06] for SOo(d, 1) relies on the use of a recursion formula 
together with the original result of [BR04] for d = 2 and the corresponding 
one for d = 3, obtained along the same lines. The formula was established 
again in [CK0P11] as the result of a comparison with invariant trilinear 
forms arising from the representation theory of real symplectic groups 
[CK0P11, Proposition 4.1]. It also falls under the general treatment of 
real-rank one groups exposed in [BKZ14]. 

Our formula (|), obtained in the case n = 2 relies on recent results of 
Fatawat and Vyas [VF14] on very well-poised hypergeometric functions. 
However, these results do not allow to treat higher order cases yet, and 
we were not able to find in the literature any useful reduction formula for 
the 6^5 function that appears in the general expression of for n> 3. 

Note that there also exist low-dimensional isogenies between the com¬ 
plex symplectic groups Sp(l,C) and Sp(2,C) and the complex orthogo¬ 
nal groups SO(3,C) and SO(5,C) respectively (see [Carl4, p.352-355] or 
[HelOl, p. 519]). This might indicate that interesting results could be 
expected for odd-dimensional complex orthogonal groups SO(2n -f 1,C), 
although the methods used here do not allow to directly address this ques¬ 
tion in full generality as of now. 

Finally, it would be interesting to extract the representation theoretic 
significance of the zeros and poles of the expression found in (|). 
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